Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Mellanox] Fix inconsistence in the shared headroom pool initialization #3057

Merged
merged 2 commits into from
Mar 15, 2024

Conversation

stephenxs
Copy link
Collaborator

What I did

Fix inconsistent initialization of the shared headroom pool by setting a small size to indicate the shared headroom pool is enabled.

Signed-off-by: Stephen Sun [email protected]

Why I did it

During initialization, if SHP is enabled

  1. the buffer pool sizes, xoff have initialized to 0, which means SHP is disabled
  2. but the buffer profiles already indicate SHP
  3. later on the buffer pool sizes are updated with off being non-zero

In case the orchagent starts handling buffer configuration between 2 and 3, it is inconsistent between buffer pools and profiles, which fails Mellanox SAI sanity check.
To avoid it, it indicates SHP enabled by setting a very small buffer pool and SHP sizes

How I verified it

Details if related

@stephenxs
Copy link
Collaborator Author

/azpw run

@mssonicbld
Copy link
Collaborator

/AzurePipelines run

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@stephenxs
Copy link
Collaborator Author

SRv6 test failed which is not relevant to my change.


    
        Args:
            table_name: The name of the table from which to fetch the keys.
            num_keys: The expected number of keys to retrieve from the table.
            polling_config: The parameters to use to poll the db.
            failure_message: The message to print if the call times out. This will only take effect
                if the PollingConfig is set to strict.
    
        Returns:
            The keys stored in the table. If no keys are found, then an empty List is returned.
        """
    
        def access_function():
            keys = self.get_keys(table_name)
            if wait_at_least_n_keys:
                return (len(keys) >= num_keys, keys)
            else:
                return (len(keys) == num_keys, keys)
    
        status, result = wait_for_result(
            access_function, self._disable_strict_polling(polling_config)
        )
    
        if not status:
            message = failure_message or (
                f"Unexpected number of keys: expected={num_keys}, received={len(result)} "
                f'({result}), table="{table_name}"'
            )
>           assert not polling_config.strict, message
E           AssertionError: Unexpected number of keys: expected=2, received=1 (('oid:0x3d000000000619',)), table="ASIC_STATE:SAI_OBJECT_TYPE_SRV6_SIDLIST"

@zhangyanzhao
Copy link
Collaborator

@bingwang-ms @neethajohn can you please help to review this PR?

@stephenxs
Copy link
Collaborator Author

/azpw run

@mssonicbld
Copy link
Collaborator

/AzurePipelines run

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@stephenxs stephenxs changed the title [Mellanox] Fix inconsistence in the shared headroom pool initialization [Mellanox] Fix inconsistent in the shared headroom pool initialization Mar 5, 2024
@stephenxs stephenxs changed the title [Mellanox] Fix inconsistent in the shared headroom pool initialization [Mellanox] Fix inconsistence in the shared headroom pool initialization Mar 5, 2024
@stephenxs stephenxs requested a review from prsunny as a code owner March 5, 2024 07:00
@prsunny prsunny merged commit a13e081 into sonic-net:master Mar 15, 2024
14 checks passed
@stephenxs stephenxs deleted the fix-inconsistent-init-shp branch March 16, 2024 00:48
mssonicbld pushed a commit to mssonicbld/sonic-swss that referenced this pull request Mar 21, 2024
…on (sonic-net#3057)

* Fix inconsistence in the shared headroom pool initialization

* Why I did it

During initialization, if SHP is enabled

the buffer pool sizes, xoff have initialized to 0, which means SHP is disabled
but the buffer profiles already indicate SHP
later on the buffer pool sizes are updated with off being non-zero
In case the orchagent starts handling buffer configuration between 2 and 3, it is inconsistent between buffer pools and profiles, which fails Mellanox SAI sanity check.
To avoid it, it indicates SHP enabled by setting a very small buffer pool and SHP sizes
@mssonicbld
Copy link
Collaborator

Cherry-pick PR to 202311: #3084

mssonicbld pushed a commit that referenced this pull request Mar 22, 2024
…on (#3057)

* Fix inconsistence in the shared headroom pool initialization

* Why I did it

During initialization, if SHP is enabled

the buffer pool sizes, xoff have initialized to 0, which means SHP is disabled
but the buffer profiles already indicate SHP
later on the buffer pool sizes are updated with off being non-zero
In case the orchagent starts handling buffer configuration between 2 and 3, it is inconsistent between buffer pools and profiles, which fails Mellanox SAI sanity check.
To avoid it, it indicates SHP enabled by setting a very small buffer pool and SHP sizes
cscarpitta pushed a commit to cscarpitta/sonic-swss that referenced this pull request Apr 5, 2024
…on (sonic-net#3057)

* Fix inconsistence in the shared headroom pool initialization

* Why I did it

During initialization, if SHP is enabled

the buffer pool sizes, xoff have initialized to 0, which means SHP is disabled
but the buffer profiles already indicate SHP
later on the buffer pool sizes are updated with off being non-zero
In case the orchagent starts handling buffer configuration between 2 and 3, it is inconsistent between buffer pools and profiles, which fails Mellanox SAI sanity check.
To avoid it, it indicates SHP enabled by setting a very small buffer pool and SHP sizes
superchild pushed a commit to superchild/sonic-swss that referenced this pull request Apr 24, 2024
* Fixes mock test failure

* Fixes mock test run failure

fixes pipeline run failure

FAIL: p4orch_tests_usan
=======================

../../../orchagent/vrforch.cpp:113:41: runtime error: member call on
null pointer of type 'struct RouteOrch'
../../../orchagent/vrforch.cpp:113:41: runtime error: member access
within null pointer of type 'struct RouteOrch'
FAIL p4orch_tests_usan (exit status: 139)

* Fixed orchagent crash in VM with the Qos BUFFER_QUEUE|system-port|Queue-id-range config (sonic-net#3050)

* Fixed orchagent crash in VM with the Qos BUFFER_QUEUE|system-port|Queue-id-range config

* [intfsorch] Enable ipv6 proxy ndp along with proxy arp (sonic-net#3045)

* [intfsorch] Enable ipv6 proxy ndp along with proxy arp

setting SAI_VLAN_ATTR_UNKNOWN_MULTICAST_FLOOD_CONTROL_TYPE to
SAI_VLAN_FLOOD_CONTROL_TYPE_NONE when proxy arp is enabled. This fixes a
bug where ipv6 NS packets were flooding ports with duplicate packets. We
now set multicast flood type to none.

* Fix multi VLAN neighbor learning (sonic-net#3049)

What I did

When adding a new neighbor, check if the neighbor IP has already been learned on a different VLAN. If it has, remove the old neighbor entry before adding the new one.

Why I did it
On Gemini devices, if a neighbor IP moves from an active port in one VLAN to a second VLAN, then back to the first VLAN (with 3 different MAC addresses), orchagent will crash. Even though the MAC address of the last move is different from the first MAC address, orchagent believes the last MAC address to already be programmed in the hardware and tries to set an attribute of the entry which doesn't exist.

* [asan] Disable the "maybe-uninitialized" warning when compiled with ASAN enabled.

* Set HOST_TX_READY_NOTIFY attribute only after query capabilities(sonic-net#3070)

*Set HOST_TX_READY_NOTIFY attribute only after query capabilities

* [EVPN] Skip EVPN routes with invalid VNI or router mac field (sonic-net#3073)

* Skip EVPN routes with invalid VNI or router mac field

* Add port flap count and last flap timestamp to APPL_DB (sonic-net#3052)

* Add port flap count and last flap timestamp

* Add basic fabric link monitoring counters and states handling. (sonic-net#2988)

* Add basic fabric link monitoring counters and states handling.

* [Mellanox] Fix inconsistence in the shared headroom pool initialization (sonic-net#3057)

* Fix inconsistence in the shared headroom pool initialization

* Why I did it

During initialization, if SHP is enabled

the buffer pool sizes, xoff have initialized to 0, which means SHP is disabled
but the buffer profiles already indicate SHP
later on the buffer pool sizes are updated with off being non-zero
In case the orchagent starts handling buffer configuration between 2 and 3, it is inconsistent between buffer pools and profiles, which fails Mellanox SAI sanity check.
To avoid it, it indicates SHP enabled by setting a very small buffer pool and SHP sizes

* [acl] Add IN_PORTS qualifier for L3 table (sonic-net#3078)

* Apply IN_PORTS qualifiier for L3 table

Why I did it
IN_PORTS qualifier was allowed for L3 table in 202012 release and below. Changes in sonic-net#1982 removed that support leading to regression in some of our testcases. The following error was observed
ERR swss#orchagent: :- validateAclRuleMatch: Match SAI_ACL_ENTRY_ATTR_FIELD_IN_PORTS in rule RULE_1 is not supported by table DATAACL

* [bulker] add support for neighbor bulking (sonic-net#2768)

Adding support for sai_neighbor_api_t bulking in bulker.h

* [buffermgrd] Move switch-statement outside of if-statement in BufferMgr::doTask (sonic-net#3055)

* [buffermgr] Moved switch statement outside of if-statmement in Buffermgr::doTask

The switch statement which would normally erase buffer events was moved
to be inside the if-statement which would only enter if the event is a
SET event. This was introduced in commit e5329c39.

This would cause an infinite loop, since non-set events would never be
erased.

The switch statement has now been moved to occur outside the if,
allowing for non-set commands to be processed.

* [portsorch] process only updated APP_DB fields when port is already   created (sonic-net#3025)

* [portsorch] process only updated APP_DB fields when port is already created

What I did

Fixing an issue when setting some port attribute in APPL_DB triggers serdes parameters to be re-programmed with port toggling. Made portsorch to handle only those attributes that were pushed to APPL_DB, so that serdes programming happens only by xcvrd's request to do so.

* [Copp]Refactor coppmgr tests (sonic-net#3093)

What I did
Refactoring coppmgr mock tests

Why I did it
After migration to bookworm, coppmgr tests started failing due to the use of sudo commands.

* Revert "[acl] Add IN_PORTS qualifier for L3 table (sonic-net#3078)" (sonic-net#3092)

This reverts commit 9d4a3ad.
*Revert "[acl] Add IN_PORTS qualifier for L3 table"

* [orchagent] TWAMP Light orchagent implementation (sonic-net#2927)

* [orchagent] TWAMP Light orchagent implementation. (sonic-net#2927)
* What I did
Implemented the TWAMP Light feature according to the SONiC TWAMP Light HLD(sonic-net/SONiC#1320).

* Clang format change. (sonic-net#3080)

What I did
This PR has no real code change. It is purely clang formatting. It only applies to the P4Orch codes.
Commands that I run:
find orchagent/p4orch -name *.h -o -name .cpp | xargs clang-format -i -style="{BasedOnStyle: Microsoft, DerivePointerAlignment: false}"

find orchagent -name response_publisher -o -name return_code.h | xargs clang-format -i -style="{BasedOnStyle: Microsoft, DerivePointerAlignment: false}"

* T2-VOQ-VS: Fix iBGP bringup issue  (sonic-net#3053)

* Fix iBGP bringup issue T2-vswitch
* On T2-VOQ chassis Emulation with multi-asic linecards, iBGP sessions dont come up. Related Issue: sonic-net/sonic-buildimage#18129

* [Fdbsyncd] Adding extern_learn flag with fdb entry so Kernel doesn't age out (sonic-net#2985)

* Adding extern_learn flag with fdb entry so that Kernel doesn't age out the MAC

* [Fdbsyncd] Adding extern_learn flag with fdb entry so Kernel doesn't age out

What I did
extern_learn flag is added while programming the fdb entry into the Kernel. This will make sure that kernel doesn't age out the fdb entry. (#15004)

How I did it
A flag extern_learn will be passed while programing the fdb entry. (#15004)

How to verify it
Tested MAC add/del to the Kernel from the local FDB entry. (#15004)

Signed-off-by: [email protected]

---------

Signed-off-by: [email protected]
Co-authored-by: Sudharsan Dhamal Gopalarathnam <[email protected]>

* Fix oper FEC retrieval after warmboot (sonic-net#3100)

Updating oper FEC status in state_db after warm-reboot as part of refresh port status call

* [EVPN]Fix fpmsyncd crash when EVPN type5 is received with bgp fib suppression enabled (sonic-net#3101)

* [EVPN]Fix fpmsyncd crash when EVPN type5 is received with bgp fib suppression enabled

* [portsorch] Handle TRANSCEIVER_INFO table on warm boot (sonic-net#3087)

* Add existing data from TRANSCEIVER_INFO table

* Introduce a new role for DPU-NPU Interconnect

Signed-off-by: Vivek Reddy Karri <[email protected]>
Co-authored-by: Sudharsan Dhamal Gopalarathnam <[email protected]>

* [p4orch] Clang format change. (sonic-net#3096)

What I did
[p4orch]  This PR has no real code change. It is purely clang formatting. 
It does the same as sonic-net#3080.

* [dash] fix ENI admin state update (sonic-net#3081)

* [dash] fix ENI admin state update

* Add force option for fabric port unisolate command (sonic-net#3089)

What I did
Add force option to the unisolate link command, so users can make the links not isolate if they want.
depends on sonic-net/sonic-buildimage#18447

* [twamporch] Explicitly initialize local variable (sonic-net#3115)

What I did
Explicitly initialized local variable.

Why I did it
We met below error message in sonic-buildimage armhf build (sonic-net/sonic-buildimage#18334)

* Add bookworm build to the PR checkers (sonic-net#3114)

What I did
Add a Bookworm build to the PR checkers. Also fix some Bookworm build errors that crept in.

Why I did it
Buildimage now builds swss for Bookworm, so the build needs to succeed.

* [ACL] Remove flex counter when updating ACL rule (sonic-net#3118)

What I did
This PR is to fix sonic-net/sonic-buildimage#18719

When ACL rule is created for the first time, a flex counter is created and registered. When the same ACL rule is being updated, the FlexCounter created before is not removed, and another FlexCounter is created and registered.

Why I did it
Fix the issue that FlexCounter is duplicated when updating existing ACL rule.

---------

Signed-off-by: [email protected]
Signed-off-by: Vivek Reddy Karri <[email protected]>
Co-authored-by: saksarav-nokia <[email protected]>
Co-authored-by: Nikola Dancejic <[email protected]>
Co-authored-by: Lawrence Lee <[email protected]>
Co-authored-by: Oleksandr Ivantsiv <[email protected]>
Co-authored-by: noaOrMlnx <[email protected]>
Co-authored-by: Lior Avramov <[email protected]>
Co-authored-by: Prince George <[email protected]>
Co-authored-by: jfeng-arista <[email protected]>
Co-authored-by: Stephen Sun <[email protected]>
Co-authored-by: Neetha John <[email protected]>
Co-authored-by: Amir <[email protected]>
Co-authored-by: Stepan Blyshchak <[email protected]>
Co-authored-by: Sudharsan Dhamal Gopalarathnam <[email protected]>
Co-authored-by: xiaodong hu <[email protected]>
Co-authored-by: mint570 <[email protected]>
Co-authored-by: Deepak Singhal <[email protected]>
Co-authored-by: KISHORE KUNAL <[email protected]>
Co-authored-by: Vivek <[email protected]>
Co-authored-by: Yakiv Huryk <[email protected]>
Co-authored-by: Saikrishna Arcot <[email protected]>
Co-authored-by: bingwang-ms <[email protected]>
@bingwang-ms
Copy link
Contributor

Hi @stephenxs, can you share the log pattern if the issue happened?

@stephenxs
Copy link
Collaborator Author

Hi @stephenxs, can you share the log pattern if the issue happened?

Hi @bingwang-ms
As the following

Feb 22 09:40:00.452958 sonic ERR syncd#SDK: :- sendApiResponse: api SAI_COMMON_API_CREATE failed in syncd mode: SAI_STATUS_INVALID_ATTRIBUTE_MAX
Feb 22 09:40:00.453055 sonic ERR syncd#SDK: :- processQuadEvent: attr: SAI_BUFFER_PROFILE_ATTR_XON_TH: 19456
Feb 22 09:40:00.453055 sonic ERR syncd#SDK: :- processQuadEvent: attr: SAI_BUFFER_PROFILE_ATTR_XOFF_TH: 33792
Feb 22 09:40:00.453132 sonic ERR syncd#SDK: :- processQuadEvent: attr: SAI_BUFFER_PROFILE_ATTR_RESERVED_BUFFER_SIZE: 19456
Feb 22 09:40:00.453132 sonic ERR syncd#SDK: :- processQuadEvent: attr: SAI_BUFFER_PROFILE_ATTR_POOL_ID: oid:0x18000000000a8f
Feb 22 09:40:00.453132 sonic ERR syncd#SDK: :- processQuadEvent: attr: SAI_BUFFER_PROFILE_ATTR_THRESHOLD_MODE: SAI_BUFFER_PROFILE_THRESHOLD_MODE_DYNAMIC
Feb 22 09:40:00.453132 sonic ERR syncd#SDK: :- processQuadEvent: attr: SAI_BUFFER_PROFILE_ATTR_SHARED_DYNAMIC_TH: 0
Feb 22 09:40:00.453221 sonic ERR swss#orchagent: :- create: create status: SAI_STATUS_INVALID_ATTRIBUTE_MAX
Feb 22 09:40:00.453236 sonic ERR swss#orchagent: :- processBufferProfile: Failed to create buffer profile pg_lossless_10000_5m_profile with type BUFFER_PROFILE_TABLE, rv:-131071
Feb 22 09:40:00.453236 sonic ERR swss#orchagent: :- handleSaiCreateStatus: Encountered failure in create operation, exiting orchagent, SAI API: SAI_API_BUFFER, status: SAI_STATUS_INVALID_ATTRIBUTE_MAX

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants